fix(installation-cleanup): don't mark live agents stale (Telegram bridge was 403ing)#696
Merged
lilyshen0722 merged 1 commit intoJul 21, 2026
Conversation
…m köprüsü 403'ledi)
Telegram köprüsü bugün her gönderimde 403 döndürdü:
"Agent token not authorized for this pod". Webhook kapanmamıştı, agent de
düşmemişti — gecelik 04:00 cron'u telegram-app kurulumunu 'stale' işaretlemişti
ve pod yetkilendirmesi status 'active' şartı arıyor. yunus bir gün önce aynı
şekilde düşmüştü; köprünün iki ucu peş peşe çürüyordu.
markStaleInstallations iki "ölü" sinyaline bakıyor ve köprüde İKİSİ DE yanlış
okuyor:
1. Süresi geçmemiş token yok. Köprünün token'ı kalıcı bir dosya token'ı
(~/.commonly/tokens/<ad>.json) ve expiresAt taşımıyor;
hasValidRuntimeToken boş expiry'yi kasten geçersiz sayıyor. Bu, terk
edilmiş artıklar için doğru ama kalıcı token'ları yanlış damgalıyor.
2. 7 gündür AgentEvent yok. Pod'a mesaj göndermek HİÇ AgentEvent üretmiyor
(ölçüldü: telegram-app event sayısı 0). Gönderim-yönlü bir entegrasyon
ne kadar trafik basarsa bassın ölü görünüyor. Diğer agent'lar yalnızca
heartbeat yaydıkları için hayatta kalıyor.
Dürüst canlılık sinyali agentRuntimeTokens[].lastUsedAt: agentRuntimeAuth onu
HER kimlik doğrulanmış istekte yazıyor. Servis o user dökümanını zaten
yüklüyordu — cevabı elinde tutup okumuyordu, dolayısıyla ek sorgu maliyeti yok.
İkinci hata, bunu ararken çıktı ve daha geniş: YENİ KURULUMLAR İÇİN YAŞ TABANI
YOKTU. dev-jr 14:31'de kuruldu, dakikalar sonra stale işaretlendi — token'ı
henüz kullanılmamış, event'i yok, çünkü kimse daha @-mention atmamıştı. Bu
normal bir durum: departmanlar iş verilmeden önce provision ediliyor ve tembel
başlayan daemon'lar mention'a kadar koşmuyor. pruneStaleInstallations 14 gün
sonra bunları SİLECEKTİ. createdAt < cutoff tabanı eklendi.
Doğrulama:
- 5 birim testi (bu servisin ilk testleri). Düzeltme geçici kaldırılınca
outage testi kırmızıya döndü — test boş değil, gerçekten koruyor.
- Gerçek veriye karşı sweeper koşusu: düzeltmeden önce marked=1 (dev-jr'ı
yanlış yakaladı), sonra marked=0.
- telegram-app + yunus canlı POST: HTTP 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lilyshen0722
approved these changes
Jul 21, 2026
lilyshen0722
left a comment
lilyshen0722
left a comment
Contributor
There was a problem hiding this comment.
Excellent diagnosis. Verified the blind-spot claim against the code: hasValidRuntimeToken's null-expiry-is-invalid stance is right for abandoned stragglers but wrong for long-lived file tokens, and message posting indeed emits no AgentEvent — so a send-only integration trips both 'dead' signals while actively relaying. lastUsedAt is the honest signal and it's already on the loaded user doc, so the fix is free. The regression test covering the exact outage case (null expiry + recent use) is exactly what this cron needed. This also protects every BYO wrapper agent using file tokens. Fourth quality PR in a row — thank you, and sorry the cron ate your bridge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly cleanup cron marked a live, actively-relaying agent installation as
stale, which made every request from it fail with403 Agent token not authorized for this pod(pod authorization requiresstatus: 'active').Hit in production on 2026-07-19: the Telegram bridge (
telegram-app) went silent.yunushad gone the same way the day before — both ends of the bridge decaying one after the other. Related: APP-47, filed 2026-07-08, which predicted this recurrence.Root cause
markStaleInstallationstreats two signals as "dead", and both misread a send-only integration:expiresAt;hasValidRuntimeTokendeliberately treats null-expiry as invalid. That's right for abandoned stragglers, but wrong for tokens that are intentionally permanent.AgentEventfor 7 days. Posting a pod message emits no AgentEvent at all (measured:telegram-appevent count was 0). A send-only integration looks dead no matter how much traffic it pushes. Other agents survive only because they emitheartbeatevents.Both "dead" signals fired for an agent that was demonstrably alive.
Fix
Use
agentRuntimeTokens[].lastUsedAtas the liveness signal —agentRuntimeAuthwrites it on every authenticated request. The service already loads that user document, so this costs no extra query; the answer was in hand and simply not read.Second bug found while fixing this
There was no age floor for new installations. A freshly provisioned agent reads "dead" on both signals purely because nobody has @-mentioned it yet — its token has never been used and it has emitted no events. That is a normal state: departments get provisioned before work is assigned, and lazily-started daemons don't run until mentioned.
Observed:
dev-jr, installed that afternoon, was marked stale minutes later.pruneStaleInstallationswould have deleted it 14 days on. AddedcreatedAt < cutoff.Verification
marked=1before the age floor (it wrongly caughtdev-jr),marked=0after.telegram-app+yunuslive POST: HTTP 200.main(6 commits ahead of where this was authored); tests still green.Note on scope
APP-47 also suggests having CLI
agent attachwrite anexpiresAt. That is not done here — if permanent tokens are an intentional choice, that item should be dropped rather than implemented. This PR fixes the sweeper's reading of liveness, not the token format.🤖 Generated with Claude Code